Gibbs Sampling for the Uninitiated

This document is intended for computer scientists who would like to try out a Markov Chain Monte Carlo (MCMC) technique, particularly in order to do inference with Bayesian models on problems related to text processing. We try to keep theory to the absolute minimum needed, and we work through the details much more explicitly than you usually see even in “introductory” explanations. That means we’ve attempted to be ridiculously explicit in our exposition and notation. After providing the reasons and reasoning behind Gibbs sampling (and at least nodding our heads in the direction of theory), we work through two applications in detail. The first is the derivation of a Gibbs sampler for Naive Bayes models, which illustrates a simple case where the math works out very cleanly and it’s possible to “integrate out” the model’s continuous parameters to build a more efficient algorithm. The second application derives the Gibbs sampler for a model that is similar to Naive Bayes, but which adds an additional latent variable. Having gone through the two examples, we discuss some practical implementation issues. We conclude with some pointers to literature that we’ve found to be somewhat more friendly to uninitiated readers.